WITP 5(01).book(WITP_A_315534.fm)

Notice and comment rulemaking is central to how U.S. federal agencies craft new regulation. E-rulemaking, the process of soliciting and considering public comments that are submitted electronically, poses a challenge for agencies. The large volume of comments received makes it difficult to distill and address the most substantive concerns of the public. This work attempts to alleviate this burden by applying existing machine learning techniques to the problem of recognizing citation sentences. A citation in this context is defined as a statement in which the author of the public comment references an external source of factual information that is associated with a specific person or organization. The problem is formulated as a binary classification problem: Is a specific person or organization mentioned in a sentence being referenced as an external source of information? We show that our definition of a citation is reproducible by human judges and that citations can be detected using machine learning techniques with some success. Casting this as a machine learning problem requires selecting an appropriate representation of the sentence. Several feature sets are evaluated individually and in combination. Superior results are obtained by combining feature sets. Syntactic features, which characterize the structure of the sentence rather than its content, significantly improve accuracy when combined with other features, but not when used in isolation. Although prediction Jaime Arguello is a Ph.D. student at the Language Technologies Institute at Carnegie Mellon University. His work focuses on text data mining, information retrieval, and natural language processing. Jamie Callan is a Professor at the Language Technologies Institute, a graduate department in Carnegie Mellon University’s School of Computer Science. His research and teaching focus on text-based information retrieval. His recent research studies advanced search engine architectures, federated search across groups of search engines, adaptive information filtering, text analysis and organization, text mining, and automatic collection of instructional materials for an intelligent reading tutor. His earlier IR research included first generation Web-search systems, integration of text search with relational database systems, and information literacy in K-12 education. Dr. Stuart W. Shulman is Director of the Sara Fine Institute in the School of Information Sciences at the University of Pittsburgh. He is also the founder and Director of the Qualitative Data Analysis Program (QDAP) at Pitt’s University Center for Social and Urban Research, which is a fee-for-service coding lab working on projects funded by the National Science Foundation, the National Institutes of Health, DARPA, and other funding agencies. He has been Principal Investigator and Project Director on related National Science Foundation–funded research projects focusing on electronic rulemaking, human language technologies, coding across the disciplines, digital citizenship, and service-learning efforts in the United States. Dr. Shulman is the Editor-in-Chief of the Journal of Information Technology & Politics. This work was supported in part by the eRulemaking project and NSF grants IIS-0240334 and IIS0429293. We are grateful to the U.S. EPA and the U.S. Department of the Interior’s FWS for providing the public comment data that made this research possible. We thank the coders at the Qualitative Data Analysis Program (QDAP) for their time and effort in producing our gold standard data and for providing the feedback that guided the evolution of our coding scheme. We also thank the anonymous reviewers for their comments. Any opinions, findings, conclusions, and recommendations expressed in this paper are the authors’ and do not necessarily reflect those of the sponsors. Address correspondence to: Jaime Arguello, Carnegie Mellon University, Pittsburgh, PA (E-mail: [email protected]). 50 JOURNAL OF INFORMATION TECHNOLOGY & POLITICS error rate is adequate, coverage could be improved. An error analysis enumerates short-term and long-term challenges that must be overcome to improve recall.